PATENT APPLICATION 

WHAT IS CLAIMED IS: 

1 1 . A video-based animal behavior analysis system, comprising: 

2 a computer configured to determine a position and shape of an animal from 

3 video images and characterize activity of said animal based on analysis of 

4 changes in said position and said shape over time. 

1 2. The system of claim 1, further comprising: 

2 a video camera and a video digitization unit coupled to said computer for 

3 capturing said video images and converting said video images from analog to 

4 digital format. 

1 3. The system of claim 2, further comprising: 

1 an animal identification, segregation, and tracking module receiving 

2 said video images. 

3 

1 4. The system of claim 3, wherein said computer further includes a 

2 behavior identification module for characterizing activity of said animal, said 

3 behavior identification module being coupled to said animal identification, 

4 segregation, and tracking module. 

1 5. The system of claim 4, wherein said computer further includes a 

2 standard animal behavior storage module that stores information about known 
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3 behavior of a predetermined standard animal for comparing the activity of said 

4 animal, said standard animal behavior storage module being coupled to said 

5 behavior identification module. 

1 6. The system of claim 1, wherein said animal is a mouse. 

1 7. The system of claim 1, wherein said animal is a rat. 

1 8. A method of determining and characterizing activity of an animal 

2 using computer processing of video images, comprising the steps of: 

3 detecting an animal in said video images; 

4 tracking changes to said animal over a plurality of said video images; 

5 identifying and classifying said changes to said animal; and 

6 characterizing said activity of said animal based on comparison to pre-trained 

7 models or rules of such activity. 

1 9. The method of claim 8, wherein said step of characterizing said 

2 activity includes the steps of: 

3 describing a sequence of postures as behavior primitives; and 

4 aggregating behavior primitives into actual behavior over a range of images; 

1 10. The method of claim 9, wherein said step of characterizing said 

2 activity by describing and aggregating behavior primitives further includes the 

3 steps of: 
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4 describing a set of conditions and rules required for characterizing said 

5 activities; and 

6 matching and testing generated features to see if said conditions and rules are 

7 satisfied; 

1 11. The method of claim 8, wherein said detecting an animal includes 

2 using a background subtraction method comprising the steps of: 

3 apply a lenient threshold on a difference between a current image and a 

4 background so as to determine a broad region of interest; 

5 classify by intensity values various pixels in said region of interest to obtain 

6 said animal, by selecting only those intensity values that belong to the set of 

7 model intensity values of the animal; and 

8 refine contours of said animal image by smoothing. 

1 12. The method of claim 10, wherein said posture determination and 

2 description includes using statistical and contour-based shape information. 

1 13. The method of claim 12, wherein said step of identifying and 

2 classifying changes to said animal includes using statistical shape information 

3 selected from the group consisting of: 

4 area of the animal; 

5 centroid position of the animal; 

6 bounding box and its aspect ratio of the animal; 

7 eccentricity of the animal; and 
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8 a directional orientation of the animal relative to an axis as generated with a 

9 Principal Component Analysis. 

1 14. The method of claim 12, wherein said step of identifying and 

2 classifying changes to said animal uses contour-based shape information 

3 selected from the group consisting of curvature measures, thickness measures, 

4 relative orientation measures, length measures, and corner points. 

1 15. The method of claim 12, wherein said step of identifying and 

2 classifying changes to said animal includes identifying a set of model postures 

3 and their description information, said set of model postures including 

4 horizontal side view posture, vertical posture, cuddled posture, horizontal 

5 front/back view posture, partially reared posture, stretched posture, hang 

6 vertical posture, hang cuddled posture, eating posture, or drinking posture. 

1 16. The method of claim 15, wherein said step of identifying and 

2 classifying changes to said animal includes classifying the statistical and 

3 contour-based shape information from a current image to assign a best- 

4 matched posture. 

1 17. The method of claim 10, wherein the said step of describing said 

2 behavior primitives includes the step of identifying patterns of postures over a 

3 sequence of images. 
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1 18. The method of claim 17, wherein said step of describing said behavior 

2 primitives step further includes the step of analyzing temporal information 

3 selected from the group consisting of direction and magnitude of movement of 

4 the centroid, increase and decrease of the eccentricity, increase and decrease of 

5 the area, increase and decrease of the aspect ratio of the bounding box, and 

6 change in contour information. 

1 19. The method of claim 10, wherein the said step of determining actual 

2 behavior by aggregating behavior primitives includes the step of analyzing 

3 temporal ordering of the primitives, such as using information about a 

4 transition from a previous behavior primitive to a next behavior primitive, and 

5 applying all applicable conditions and rules. 

1 20. The method of claim 19, wherein said temporal analysis is a time- 

2 series analysis such as Hidden Markov Model (HMMs). 

1 21. The method of claim 19, wherein the said step of determining actual 

2 behavior includes identifying actual behavior selected from a group of pre- 

3 trained behavior models. 

1 22. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of rearing up to a fully reared up or partially reared up 

3 position, and said rearing behavior is determined by the sequence of posture 

4 starting from cuddled, horizontal side-view, or horizontal front/back view 
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5 postures to ending in a vertical or partially reared posture; 

1 23. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of coming down from a reared up or partially reared up 

3 position, and said come down behavior is determined by the sequence of 

4 postures starting from vertical or partially reared postures to ending in a 

5 cuddled, horizontal side view or horizontal front/back view posture; 

1 24. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of eating, and said eating behavior is determined by a 

3 sequence of eating postures where the mouth of the animal is in touch with a 

4 food container; 

1 25. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of drinking, and said drinking behavior is determined by 

3 a sequence of drinking postures where the mouth of the animal is in touch with 

4 a water spout; 

1 26. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of digging, and said digging behavior is determined by 

3 the aft movement of bedding by the animal with its fore and hind limbs; 

1 27. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of foraging, and said foraging behavior is determined by 
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3 the movement of bedding using the mouth and forelimbs; 

1 28. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of jumping, and said jumping behavior is determined by 

3 a single up and down movement of the animal; 

1 29. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of jumping repetitively and said repetitive jumping 

3 behavior is determined by several continuous up and down movement of the 

4 animal; 

1 30. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of sniffing, and said sniffing behavior is determined by 

3 random brisk movement of the head while the rest of the body remains 

4 stationary; 

1 31. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of hanging from the top of the cage, and said hanging 
3. behavior is determined by a sequence of postures starting from vertical posture 
4 to ending in a hang vertical or hang cuddled posture; 

1 32. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of landing after hanging, and said landing behavior is 

3 determined by a sequence of postures starting from a hang vertical or hang 
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4 cuddled posture to ending in a vertical posture; 

1 33. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of sleeping, and said sleeping behavior is determined by 

3 the absence of major movements of the contour of the animal for a prolonged 

4 period of time; 

1 34. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of twitching during sleep, and said twitch behavior is 

3 determined by the detection of a brief period of substantial movement and the 

4 resumption of sleep activity; 

1 35. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of awakening from sleep, and said awaken behavior is 

3 determined by a prolonged movement of the animal after sleep has set in; 

1 36. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of grooming, and said grooming behavior determined by 

3 brisk movement of limbs and mouth in a cyclical and periodic pattern; 

1 37. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of pausing briefly, and said pause behavior is 

3 determined by brief absence of movement of the animal; 
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1 38. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of urinating, and said urinate behavior is determined by 

3 the detection of the tail being raised up and the animal remaining stationary 

4 briefly; 

1 39. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of turning, and said turn behavior is determined by a 

3 sequence of postures starting from horizontal side view or cuddled posture to 

4 ending in a horizontal front/back view posture, and vice versa; 

1 40. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of circling, and said circling behavior is determined by a 

3 3 or more successive turns; 

1 41. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of walking or running, and said walking or running 

3 behavior is determined by the continuous sideways movement of the centroid 

4 of the animal; 

1 42. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of stretching its body vertically or horizontally, and said 

3 stretch behavior is determined by a concave shape of the animal's back; 



1 43. The method of claim 21, wherein said group of behavior models 
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2 includes the behavior of chewing, and said chewing behavior is determined by 

3 the movement of the mouth while the mouth is not in touch with a food 

4 container; 

1 44. The method of claim 21, wherein said group of behavior models 

2 includes the behavior of remaining stationary, and said stationary behavior is 

3 determined by the animal remaining in the same place and not performing any 

4 of the other behaviors; 

1 45. The method of claim 21, wherein if the activity cannot be characterized 

2 by any of the behavior models, the behavior is deemed to be unknown; 

1 46. The method of claim 8, wherein said steps are also performed in night 

2 conditions by using red light to simulate such night conditions, or by using 

3 infra-red cameras to capture the images with no light; 

1 47. The method of claim 8, wherein said steps are also performed with a 

2 plurality of cages or arenas, each of which contains a single animal; 

1 48. The method of claim 8, wherein said step of detecting animal includes 

2 the step of detecting body parts of the animal; 

1 49. The method of claim 48, wherein said body parts include the head; 
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1 50. The method of claim 48, wherein said body parts include the tail; 

1 51. The method of claim 48, wherein said body parts include the ear; 

1 52. The method of claim 48, wherein said body parts include the upper and 

2 lower back; 

1 53. The method of claim 48, wherein said body parts include the abdomen; 

1 54. The method of claim 48, wherein said body parts include the hind- 

2 limbs; 

1 55. The method of claim 48, wherein said body parts include the 

2 forelimbs; 
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